
---
title: "Analysis of Müller and Bostrom 2016"
author: "Allan Dafoe"
date: "September 26, 2016"
output:
  pdf_document: default
  html_document:
    number_sections: yes
---




Analysis for Allan Dafoe and Stuart Russell "Yes. A response to Etzioni's article, 'Are the Experts Worried About the Existential Risk of Artificial Intelligence?'"

All files related to this analysis can be found on dataverse here:  
http://dx.doi.org/10.7910/DVN/JHR1GX

Müller and Bostrom paper is here: www.nickbostrom.com/papers/survey.pdf 
 
Müller and Bostrom data is here: http://www.pt-ai.org/polls/experts

Research Assistance by Baobao Zhang

```{r, echo=FALSE, warning=FALSE}
rm(list = ls())
set.seed(1234)
library(ggplot2)
library(grid)
```

```{r, echo = FALSE}
# new geom object to plot 
ggname <- function(prefix, grob) {
  grob$name <- grobName(grob, prefix)
  grob
}

geom_ribbon_horz <- function(mapping = NULL, data = NULL,
                             stat = "identity", position = "identity",
                             ...,
                             na.rm = FALSE,
                             show.legend = NA,
                             inherit.aes = TRUE) {
  layer(
    data = data,
    mapping = mapping,
    stat = stat,
    geom = GeomRibbon_horz,
    position = position,
    show.legend = show.legend,
    inherit.aes = inherit.aes,
    params = list(na.rm = na.rm,
                  ...)
  )
}

GeomRibbon_horz <- ggproto(
  "GeomRibbon_horz", Geom,
  default_aes = aes(
    colour = NA, fill = "grey20", size = 0.5, linetype = 1,
    alpha = NA
  ),
  
  required_aes = c("y", "xmin", "xmax"),
  
  draw_key = draw_key_polygon,
  
  handle_na = function(data, params) {
    data
  },
  
  draw_group = function(data, panel_scales, coord, na.rm = FALSE) {
    if (na.rm)
      data <- data[stats::complete.cases(data[c("y", "xmin", "xmax")]),]
    data <-
      data[order(data$group, data$x),]
    
    # Check that aesthetics are constant
    aes <-
      unique(data[c("colour", "fill", "size", "linetype", "alpha")])
    if (nrow(aes) > 1) {
      stop("Aesthetics can not vary with a ribbon")
    }
    aes <- as.list(aes)
    
    # Instead of removing NA values from the data and plotting a single
    # polygon, we want to "stop" plotting the polygon whenever we're
    # missing values and "start" a new polygon as soon as we have new
    # values.  We do this by creating an id vector for polygonGrob that
    # has distinct polygon numbers for sequences of non-NA values and NA
    # for NA values in the original data.  Example: c(NA, 2, 2, 2, NA, NA,
    # 4, 4, 4, NA)
    missing_pos <-
      !stats::complete.cases(data[c("y", "xmin", "xmax")])
    ids <- cumsum(missing_pos) + 1
    ids[missing_pos] <- NA
    
    positions <- plyr::summarise(
      data,
      y = c(y, rev(y)), x = c(xmax, rev(xmin)), id = c(ids, rev(ids))
    )
    munched <-
      coord_munch(coord, positions, panel_scales)
    
    ggname(
      "geom_ribbon_horz", polygonGrob(
        munched$x, munched$y, id = munched$id,
        default.units = "native",
        gp = gpar(
          fill = alpha(aes$fill, aes$alpha),
          col = aes$colour,
          lwd = aes$size * .pt,
          lty = aes$linetype
        )
      )
    )
  }
)
```

```{r, echo=FALSE, results='hide'}
## Importing and Processing Data

d <- read.csv("wfr_export.csv")

#To isolate the Top 100, subset the data on those who replied on May 29, 2013 or later. 
#This follows from the fact that Müller and Bostrom (2016) report that the Top 100 Survey started May 2013 and that they received 29 responses. The public dataset has exactly 29 responses starting from then. Consistent with this is that the respondents then report having especially high levels of expertise in "technical work in AI". (h/t Carrick Flynn)

d$date <- as.Date(d$Submission.Date,"%m/%d/%Y")
d$top100 <- d$date >= "0013-05-29"
d <- d[!is.na(d$top100),]

# Confirming that there is only 29
sum(d$top100)

# Replace never with max value, to get correct quantiles
d$Year.10.[is.na(d$Year.10.)] <- as.integer(max(d$Year.10., na.rm=T))
d$Year.50.[is.na(d$Year.50.)] <- as.integer(max(d$Year.50., na.rm=T))
d$Year.90.[is.na(d$Year.90.)] <- as.integer(max(d$Year.90., na.rm=T))

r <- d$top100 
```


# Timelines

## Etzioni's Results
Etzioni asked his respondents the following:   
"In his book, Nick Bostrom has defined Superintelligence as ‘an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.’ When do you think we will achieve Superintelligence?"

Etzioni then provided the experts with four answer options: "In the next 10 years" (0% selected this), "In the next 10-25 years" (7.5%), "In more than 25 years" (67.5%), and "Never" (25%). 

## Commentary on Etzioni's Results

From this Etzioni draws his main inference that "In essence, according to 92.5 percent of the respondents, superintelligence is beyond the foreseeable horizon." 

As we discuss, we think there are problems with this interpretation. First, and most importantly, it declares >25 years beyond the foreseeable horizon and thus as not worth worrying about. By that logic we should not worry about climate change, retirement, etc...  

Second, we see that 7.5% of these experts actually thought that _superintelligence_ would be achieved within 25 years. Given that superintelligence would be a world changing achievement, that 7.5% think it will arrive so soon is noteworthy. 

Third, though Etzioni's question seems intuitively straightforward, it is actually deeply ambiguous. Should the reader interpret this as asking them by when they believe there is a greater than 50% chance?  A greater than 90% chance? A greater than 99% chance?  Or a 100% chance? As we know from theory and empirics, these different questions ask about very different points on a person’s probability distribution over the future, and lead to answers that differ on the order of decades. 

## Timeline for Müller and Bostrom's Top 100 AI researchers

Müller and Bostrom (2016) surveyed the top 100 cited AI researchers; 29 responded. 

They ask the Top100 the following:

"Define a 'high–level machine intelligence' (HLMI) as one that can carry out most human professions at least as well as a typical human."

"For the purposes of this question, assume that human scientific activity continues without major negative disruption. By what year would you see a (10% / 50% / 90%) probability for such HLMI to exist?"  

The descriptive statistics for their answers are as follows, written in order of 10%, 50%, and 90%:
```{r}
summary(d$Year.10.[r])
summary(d$Year.50.[r])
summary(d$Year.90.[r])
```

We can see these responses using a plot of the cumulative distribution function for the beliefs of the Top 100 about High Level Machine Intelligence existing, over time. 

```{r, echo=FALSE, fig.width=10}
p10 <- c(.1, summary(d$Year.10.[r])[c(2,3, 5)]-2016)
p50 <- c(.5, summary(d$Year.50.[r])[c(2,3, 5)]-2016)
p90 <- c(.9, summary(d$Year.90.[r])[c(2,3,5)]-2016)

res_tab <- data.frame(rbind(p10,p50,p90))
names(res_tab) <- c("probs", "lower", "median", "upper")

hb_plot <- ggplot(res_tab, aes(x = median, y = probs, xmin = lower, xmax = upper)) +
  geom_line(color = "darkblue") + geom_point(color = "darkblue") + 
  scale_y_continuous(labels = scales::percent, limits = c(0,1)) +
  xlab("Years from 2016") + ylab("Percent Probability") + coord_cartesian(xlim=c(0, 80))+
  geom_ribbon_horz(alpha = 0.3, fill = "blue") + theme_bw() + ggtitle("Beliefs of AI Top 100 about HLMI, 50% Credible Interval") + geom_vline(xintercept=25, col="red") + annotate("text", x=23,  y=.9, hjust=1, label="Etzioni's \n 'Foreseable Horizon'", col="red")
hb_plot  
ggsave("overall_prediction_top100.pdf", width=8, height=4, dpi = 300)
ggsave("overall_prediction_top100.png", width=8, height=4, dpi = 300)
```

The blue line depicts the median belief. Left edge of the “credible interval” depicts the beliefs of the 25th percentile, right edge the 75th percentile. If each expert is as likely to be correct as another, we can interpret this figure as a 50% credible interval for the arrival time of HLMI. The red line depicts Etzioni's “Foreseeable Horizon”, beyond which he states we need not worry. 



## Comparison Between Etzioni's, and Müller and Bostrom's Results
Etzioni found that 92.5% of respondents selected >25 years for the arrival of superintelligence. As is clear from the above figure, the Top 100 also put a lot of probability mass on >25 years. Specifically, we know that `r round(sum(d$Year.50.[r]>=(2016+25))/length(d$Year.50.[r]),2)*100`% of the sample was at least 50% confident that HLMI would not arrive within 25 years, and  `r round(sum(d$Year.90.[r]>=(2016+25))/length(d$Year.90.[r]),2)*100`% of the sample was at least 10% confident that HLMI would not arrive within 25 years. Given that HLMI is a much lower technical bar than superintelligence, these numbers are entirely consistent with Etzioni's results. (This is the basis for our claim that "more than half of Bostrom's respondents gave dates beyond 25 years for a mere 50% probability of achieving mere human-level intelligence.")

Etzioni reported that 25% of his sample selected "never" for when superintelligence will be achieved. Similarly, `r round(sum(d$Year.90.[r]>(2016+300))/length(d$Year.90.[r]),2)*100`% of Müller and Bostrom Top 100 thought there was at least a 10% chance that HLMI would not have been achieved within 300 years. If respondent's interpreted Etzioni's question asking about when **we will achieve** superintelligence as asking for at least 90% confidence, then we see that Müller and Bostrom's results are entirely consistent with Etzioni's. 


# Is Superintelligence a Threat to Humanity?

Etzioni concluded that "few believe AI is a threat to humanity", though he did not ask any questions about this. 

To provide data on this question, we turn to the following question asked by Müller and Bostrom:

"Assume for the purpose of this question that such HLMI will at some point exist. How positive or negative would be overall impact on humanity, in the long run? Please indicate a probability for each option. (The sum should be equal to 100%.)” 

The five options were: “Extremely good – On balance good – More or less neutral – On balance bad – Extremely bad (existential catastrophe)"

```{r, echo=FALSE, results='hide'}
remove.NA <- function(var){ 
  var[is.na(var)] <- 0 
  return(var)
}

r <- d$top100

d$good2 <- remove.NA(d$Extremely.Good) 
d$good1 <- remove.NA(d$On.balance.good) 
d$good0 <- remove.NA(d$More.or.less.neutral) 
d$goodm1 <- remove.NA(d$On.balance.bad) 
d$goodm2 <- remove.NA(d$Extremely.bad..existential.catastrophe.) 

summary(d$good2[r])
summary(d$good1[r])
summary(d$good0[r])
summary(d$goodm1[r])
summary(d$goodm2[r])

y <- c(median(d$good2[r]), median(d$good1[r]), median(d$good0[r]), median(d$goodm1[r]), median(d$goodm2[r]))
xlab <- c("Extremely good", "On balance good", "More or less neutral", "On balance bad", "Extremely bad \n (existential catastrophe)")

bad.v.bad <- d$goodm1 + d$goodm2
summary(bad.v.bad[r])
```

```{r, echo=FALSE, fig.width=10}
barplot(y, names.arg=xlab, cex.names=0.8, main="Median Beliefs of Top 100 About Outcome of HLMI")

```

Thus, most of the Top 100 experts thought that high level machine intelligence is likely to be good or very good for humanity, as do we, moreso to the extent that these issues are taken seriously. 
<!-- But at least half of the Top 100 thought there was at least a `r median(d$goodm1[r])`% chance that HLMI would be “on balance bad”, and at least a `r median(d$goodm2[r])`% chance that it would be "extremely bad (existential catastrophe)." --> 
At least half of the Top 100 thought there was at least a `r round(median(bad.v.bad[r]),0)`% chance of an "on balance bad" or "extremely bad (existential catastrophe)" outcome. A 2016 (unpublished) survey of recently published AI researchers has found similar results (with a 5% median prediction on existential catastrophe). 


# Misc 

We should also clear up one apparent misunderstanding regarding the survey populations. Etzioni says that Bostrom conducted “four different surveys of groups such as participants in a conference called ‘Philosophy and Theory of AI’ … and members of the Greek Association for Artificial Intelligence.” Etzioni's own survey, in contrast, was of Fellows of AAAI, described as “leading researchers in the field … who are recognized as having made significant, sustained contributions.” Inexplicably, he neglects to mention that Bostrom’s fourth survey was of the top 100 most cited living AI researchers, or that the results of all four surveys were largely similar.
